Image pick-up device and image pick-up method adapted with image pick-up sensitivity

ABSTRACT

In an image pick-up apparatus including an image pick-up unit comprised of color filters having different spectral characteristics, and serving to pick up image of object, there are provided an adjustment unit for adjusting color reproduction value and noise value representing noise feeling, a matrix coefficient determination unit for determining matrix coefficients on the basis of adjustment of the adjustment unit, and a matrix transform processing unit for performing matrix transform processing with respect to image which has been picked up at the image pick-up device unit on the basis of the matrix coefficients.

TECHNICAL FIELD

The present invention relates to an image pick-up apparatus and an imagepick-up method which are adapted for picking up image of object, andmore particularly to an image pick-up apparatus and an image pick-upmethod which are adapted for performing image pick-up operation in amanner suitable with respect to image pick-up sensitivity.

Further, this Application claims priority of Japanese Patent ApplicationNo. 2002-375423, filed on Dec. 25, 2002, the entirety of which isincorporated by reference herein.

BACKGROUND ART

In recent years, image pickup apparatuses (digital cameras, and/or colorscanners, etc.) and image processing softwares which are oriented toconsumer have been popularized. Users who themselves edit, by imageprocessing software, image generated by picking up an image of object bythe image pick-up apparatus are being increased. Moreover, request withrespect to picture quality of image which has been picked up by theimage pick-up apparatus has become strong. Ratio of users who mentionthat picture quality is high as main condition in purchasing the imagepick-up apparatus is being increased. Here, general configuration of theimage pick-up apparatus will be described below.

In the image pick-up apparatus, there is used, e.g., color filter 1 forthree primary colors of RGB as shown in FIG. 1. In this example, colorfilter 1 is constituted by the so-called Bayer arrangement with fourfilters in total of two G filters serving to allow only light of green(G) to be transmitted therethrough, one R filter serving to allow onlylight of red (R) to be transmitted therethrough and one B filter servingto allow only light of blue (B) to be transmitted therethrough being asminimum unit as indicated by single dotted lines of FIG. 1.

FIG. 2 is a block diagram showing a configuration example of a signalprocessing unit 11 for implementing various processing to RGB signalswhich have been acquired by CCDs (Charge Coupled Devices) having the RGBcolor filter 1.

An offset correction processing unit 21 serves to remove offsetcomponent included in an image signal which has been delivered from afront end 13 for implementing a predetermined processing to signalswhich have been acquired by the CCD image pick-up devices to output animage signal thus obtained to a white balance correction processing unit22. The white balance correction processing unit 22 serves to correctbalance of respective colors on the basis of color temperature of theimage signal which has been delivered from the offset correctionprocessing unit 21 and differences of sensitivities of respectivefilters of the color filter 1. The color signal acquired after undergonecorrection by the white balance correction processing unit 22 isoutputted to a gamma (γ)-correction processing unit 23. Thegamma-correction processing unit 23 performs gamma-correction withrespect to a signal which has been delivered from the white balancecorrection processing unit 22 to output a signal thus acquired to avertical direction simultaneous-izing (synchronizing) processing unit24. At the vertical direction simultaneous-izing processing unit 24,delay elements are provided. Shifts of times in vertical direction ofsignal which have been delivered from the gamma-correction processingunit 23 are simultaneous-ized (synchronized).

A RGB signal generation processing unit 25 performs interpolationprocessing for interpolating color signal delivered from the verticaldirection simultaneous-izing processing unit 24 into phase of the samespace, noise rejection (removal) processing for rejecting or removingnoise component of signal, filtering processing for limiting signalband, and high frequency band correction processing for correcting highfrequency band component of the signal band, etc. to output RGB signalsthus obtained to a luminance signal generation processing unit 26, and acolor difference signal generation processing unit 27.

The luminance signal generation processing unit 26 serves to synthesize,at a predetermined synthesis ratio, RGB signals which have beendelivered from the RGB signal generation processing unit 25 to generatea luminance signal (Y). The color difference signal generationprocessing unit 27 serves to similarly synthesize, at a predeterminedsynthesis ratio, RGB signals which have been delivered from the RGBsignal generation processing unit 25 to generate color differencesignals (Cb, Cr). The luminance signal (Y) generated by the luminancesignal generation processing unit 26 and the color difference signals(Cb, Cr) generated by the color difference signal generation processingunit 27 are outputted to a monitor provided at the outside of the signalprocessing unit 11, for example.

In a manner as stated above, it is generally performed to implementgamma-processing to an original signal and to conduct thereafter imageprocessing (linear matrix processing) by linear transform processing.

In the image pick-up apparatus as described above, there are instanceswhere since when image of object is picked up to generate image, way ofseeing varies depending upon visual environment at the time ofobservation, reproduction into desired color may not be performed. Thisis the phenomenon taking place in the case where color renderingcharacteristics of light source at the time of image pick-up operation(hereinafter referred to as image pick-up light source) and light sourceat the time of observation (hereinafter referred to an observation lightsource) are greatly different from each other. In view of the above,there is proposed, in the Japanese Patent Application Laid Open No.2002-142231 publication, etc., a method of satisfactorily performingcolor reproduction even in the case where image is reproduced by theobservation light source having color rendering characteristic differentfrom that of the image pick-up light source. Moreover, characteristicsof spectral sensitivities are shown in FIGS. 3 and 4. Curve L1 of FIG. 3indicates spectral sensitivity of R, curve L2 indicates spectralsensitivity of G, and curve L3 indicates spectral sensitivity of B.Further, curve L11 of FIG. 4 indicates spectral sensitivity of R, curveL12 indicates spectral sensitivity of G, and curve L13 indicatesspectral sensitivity of B.

On the other hand, in the image pick-up apparatus as described above,addition of the linear matrix processing is performed, and addition ofcolor filter of the image pick-up device is further performed, therebymaking it possible to dramatically improve color reproductioncharacteristic (reproducibility). In this instance, when coefficients ofa linear matrix used is determined so that color difference is simplyminimized, there are instances where noise may be increased as comparedto the prior art.

DISCLOSURE OF THE INVENTION

An object of the present invention is to provide an image pick-upapparatus and an image pick-up method which can perform linear matrixprocessing using coefficients in which color reproduction characteristicand noise reduction characteristic are taken into consideration independency upon image pick environment and/or image pick-up condition,etc.

The image pick-up apparatus according to the present invention isdirected to an image pick-up apparatus including an image pick-up unitcomprised of color filters having different spectral characteristics andserving to pick up image of object, which comprises: adjustment meansfor adjusting color reproduction value and noise value representingnoise feeling; matrix coefficient determination means for determiningmatrix coefficients on the basis of adjustment of the adjustment means;and matrix transform processing means for performing matrix transformprocessing with respect to an image which has been picked up at theimage pick-up device unit on the basis of the matrix coefficients.

In addition, the image pick-up method according to the present inventionis directed to an image pick-up method of picking up an image of objectby an image pick-up apparatus including an image pick-up unit comprisedof color filters having different spectral characteristics and servingto pick up image of object, which includes: a first step of adjustingcolor reproduction value and noise value representing noise feeling; asecond step of determining matrix coefficients on the basis ofadjustment of the first step; and a third step of performing matrixtransform processing with respect to an image which has been picked upat the image pick-up unit on the basis of the matrix coefficients.

Still further objects of the present invention and practical meritsobtained by the present invention will become more apparent from thedescription of the embodiments which will be given below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view showing an example of conventional RGB color filter.

FIG. 2 is a block diagram showing a configuration example of signalprocessing unit provided in a conventional image pick-up apparatus.

FIG. 3 is a view showing an example of spectral sensitivitycharacteristic.

FIG. 4 is a view showing another example of spectral sensitivitycharacteristic.

FIG. 5 is a block diagram showing a configuration example of an imagepick-up apparatus to which the present invention is applied.

FIG. 6 is a view showing an example of color filter for four colorsprovided in the image pick-up apparatus to which the present inventionis applied.

FIG. 7 is a view showing an example of spectral luminous efficacy curve.

FIG. 8 is a view showing feature of evaluation coefficients.

FIG. 9 is a block diagram showing a configuration example of camerasystem LSI that the image pick-up apparatus to which the presentinvention is applied has.

FIG. 10 is a block diagram showing a configuration example of signalprocessing unit of FIG. 9.

FIG. 11 is a flowchart for explaining preparation processing of imageprocessing unit.

FIG.12 is a flowchart for explaining the detail of processing fordetermining color filter for four colors of step S1 of FIG. 11.

FIG. 13 is a view showing an example of virtual curve.

FIGS. 14(A) to 14(C) are views showing examples of UMG value everyfilter.

FIG. 15 is a view showing an example of spectral sensitivitycharacteristic of color filter for four colors.

FIG. 16 is a flowchart for explaining the detail of linear matrixdetermination processing of step S2 of FIG. 11.

FIG. 17 is a view showing an example of evaluation result of colordifference.

FIG. 18 is a view showing chromaticity of a predetermined object bycolor filter for four colors.

FIG. 19 is a view showing another example of color filter for fourcolors provided in the image pick-up apparatus to which the presentinvention is applied.

FIG. 20 is a flowchart showing that linear matrix coefficients M areadaptively determined.

FIG. 21 is a view showing the state of change of noise reductioncharacteristic index when color reproduction characteristic index ischanged.

FIG. 22 is a view showing an example of spectral sensitivitycharacteristic of color filter for four colors.

FIG. 23 is a view showing histogram of image.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 5 is a block diagram showing a configuration example of an imagepickup apparatus to which the present invention is applied.

In the image pick-up apparatus shown in FIG. 5, a color filter fordiscriminating between four kinds of colors (rays of light) is providedat the front face (the face opposite to lens 42) of an image sensor 45comprised of CCDs (Charge Coupled Devices), etc. It is to be noted that,in the image pick-up apparatus, the color filter provided at the imagesensor 45 of FIG. 5 is caused to be a filter 61 for four colors shown inFIG. 6.

The color filter 61 for four colors is constituted with four filters intotal of R filter for allowing only red light to be transmittedtherethrough, B filter for allowing only blue light to be transmittedtherethrough, G1 filter for allowing only green light of the firstwavelength band to be transmitted therethrough, and G2 filter forallowing green light of the second wavelength band, which has highcorrelation with respect to the G1 filter, to be transmittedtherethrough being as the minimum unit as indicated by single dottedlines of FIG. 6. Moreover, the G1 filter and the G2 filter are arrangedat positions diagonal to each other within the minimum unit thereof.

As described in detail later, the number of kinds of colors of imageacquired by the image sensor 45 is caused to be four, and colorinformation to be acquired is increased, thereby making it possible tomore precisely represent color as compared to the case where only threekinds of colors (RGB) are acquired. Further, it becomes possible toimprove the fact that color seen as different color by the eye isreproduced into different color, and color seen as the same color isreproduced into the same color (“discriminating characteristic ofcolor”).

It is to noted that the eye of the human being is sensitive to luminanceas understood from spectral luminous efficacy curve shown in FIG. 7.Accordingly, in the color filter 61 for four colors shown in FIG. 6,there is supplemented color filter of G2 having spectral sensitivitycharacteristic close to spectral luminous efficacy curve in which moreprecise luminance information is acquired so that gradation of luminancecan be raised, and image close to way of seeing of the eye can bereproduced (there is supplemented green G2 filter newly determined withrespect to filters of R, G1, B corresponding to R, G, B of FIG. 1).

Moreover, as filter evaluation coefficient used in determining colorfilter 61 for colors, there is used UMG (Unified Measure of Goodness)which is coefficient in which, e.g., both “color reproductioncharcteristic” and “noise reduction characteristic” are taken intoconsideration.

In the evaluation using UMG, its evaluation value does not become highin the case where filter to be evaluated simply satisfies the routercondition, but overlap of spectral sensitivity distributions ofrespective filters is also taken into consideration. Accordingly, noisecan be more reduced as compared to the case of the color filterevaluated by utilizing q-factor, μ-factor or FOM (Figure of Merit).Namely, by evaluation using UMG, there is selected filter in whichspectral sensitivity characteristics of respective filters overlap witheach other to a certain degree, but substantially all characteristics donot overlap as in the case of the characteristic of R and thecharacteristic of G of FIG. 4. For this reason, even in the case whererespective color signals are amplified for the purpose of separation ofcolors, it is unnecessary to allow amplification factor to be so large.Followed by this, it is suppressed that noise component is amplified.

FIG. 8 is a view showing feature of respective filter evaluationcoefficients. It is to be noted that FIG. 8 shows the matter relating tothe number of filters which can be evaluated at a time, whether or notspectral reflection factor of object is taken into consideration, andwhether or not reduction of noise is taken into consideration withrespect to respective evaluation coefficients.

As shown in FIG. 8, q-factor indicates that the number of filters whichcan be evaluated at a time is only “one”, and spectral reflection factorof object and reduction of noise are not taken into consideration.Moreover, μ-factor indicates that while plural filters can be evaluatedat a time, spectral reflection factor of object and reduction of noiseare not taken into consideration. Further, FOM indicates that whileplural filters can be evaluated at a time and spectral reflection factorof object is taken into consideration, reduction of noise is not takeninto consideration.

On the contrary, UMG used in determining color filter 61 for four colorsindicates that plural filters can be evaluated at a time, and spectralreflection factor of object is taken into consideration and reduction ofnoise is also taken into consideration.

It is to be noted that the detail of q-factor is disclosed in ‘H. E. J.Neugebauer “Quality Factor for Filters Whose Spectral Transmittances areDifferent from Color Mixture Curves, and Its Application to ColorPhotography” JOURNAL OF THE OPTICAL SOCIETY OF AMERICA, VOLUME 46,NUMBER 10’, and the detail of μ-factor is disclosed in ‘P. L. Vora andH. J. Trussell, “Measure of Goodness of a set of color-scanningfilters”, JOURNAL OF THE OPTICAL SOCIETY OF AMERICA, VOLUME 10, NUMBER7’. Moreover, the detail of FOM is disclosed in ‘G. Sharma and H. J.Trussell, “Figures of Merit for Color Scanners, IEEE TRANSACTION ONIMAGE PROCESSING, VOLUME 6’, and the detail of UMG is disclosed in ‘S.Quan, N. Ohta, and N. Katoh, “Optimal Design of Camera SpectralSensitivity Functions Based on Practical Filter Components”, CIC, 2001’.

Returning to the explanation of FIG. 5, the microcomputer 41 controlsthe entire operation in accordance with predetermined control programs.For example, the microcomputer 41 performs exposure control by an iris43, opening/closing control of a shutter 44, control of electronicshutter of a TG (Timing Generator) 46, gain control at a front end 47,mode control of a camera system LSI (Large Scale Integrated Circuit) 48,and parameter control, etc.

The iris 43 adjusts passing of light (iris) converged by a lens 42 tocontrol light quantity which is taken thereinto by image sensor 45. Theshutter 44 controls passing of light which has been converged by thelens 42 on the basis of instruction of the microcomputer 41.

The image sensor 45 further includes image pick-up device comprised ofCCDs or CMOSs (Complementary Metal Oxide Semiconductors), and serves toconvert rays of light incident through color filter 61 for four colorsformed at the front face of the image pick-up device into electricsignals to output four kinds of color signals (R signal, G1 signal, G2signal, B signal) to the front end 47. At the image sensor 45, colorfilter 61 for four colors of FIG. 6 is provided, so that from rays oflight incident through the lens 42, components of wavelengths ofrespective bands of R, G1, G2, B are extracted. It is to be noted thatthe detail thereof will be described later with reference to FIG. 15.

The front end 47 implements correlating double sampling processing forremoving noise component, gain control processing and digital conversionprocessing, etc. to color signal delivered from the image sensor 45. Theimage data obtained after undergone various processing by the front end47 is outputted to the camera system LSI 48.

As described in detail later, the camera system LSI 48 performs variousprocessing with respect to the image data delivered from the front end47 to generate, e.g., luminance signal and color signal, to output thosesignals to image monitor 50, and to allow the image monitor 50 todisplay image corresponding to the signals.

An image memory 49 is comprised of, e.g., DRAM (Dynamic Random AccessMemory), or SDRAM (Synchronous Dynamic Random Access Memory), etc., andis suitably utilized when the camera system LSI 48 performs variousprocessing. An external memory medium 51 comprised of semiconductormemory or disc, etc. is detachably constituted with respect to, e.g.,the image pick-up apparatus of FIG. 5, so that image data compressed byJPEG (Joint Photographic Expert Group) format by the camera system LSI48 are stored thereinto.

The image monitor 50 is comprised of, e.g., LCD (liquid CrystalDisplay), etc., and serves to display picked-up images and/or variousmenu pictures, etc.

FIG. 9 is a block diagram showing a configuration example of the camerasystem LSI 48 shown in FIG. 5. Respective blocks constituting the camerasystem LSI 48 are controlled by the microcomputer 41 shown in FIG. 5through a microcomputer interface (I/F) 73.

A signal processing unit 71 performs various processing such asinterpolation processing, filtering processing, matrix operationprocessing, luminance signal generation processing and/or colordifference signal generation processing, etc. with respect to four kindsof color information delivered from the front end 47 to output, e.g.,generated image signal to the image monitor 50 through a monitorinterface 77.

An image detecting unit 72 performs detection processing such asauto-focus, auto-exposure, and/or auto-white balance, etc. on the basisof output of the front end 47 to output the result thereof to themicrocomputer 41 as occasion demands.

A memory controller 75 controls transmission/reception of data betweenprocessing blocks mutually, or transmission/reception of data between apredetermined processing block and the image memory 49 to output, e.g.,image data delivered from the signal processing unit 71 to the imagememory 49 through the memory interface 74 and to allow the image memory49 to store such image data.

An image compression/decompression (extraction) unit 76 compresses,e.g., image data delivered from the signal processing unit 71 by theJPEG format to output data thus obtained to the external memory medium51 through the microcomputer interface 73 to allow the external memorymedium 51 to store such data. The image compression/decompression unit76 further decompresses or extracts (expands) the compressed data whichhas been read out from the external memory medium 51 to output the datathus obtained to the image monitor 50 through the monitor interface 77.

FIG. 10 is a block diagram showing the detailed configuration example ofthe signal processing unit 71 shown in FIG. 9. Respective blocksconstituting the signal processing unit 71 are controlled by themicrocomputer 41 through the microcomputer interface 73.

An offset correction processing unit 91 removes noise component (offsetcomponent) included in image signal delivered from the front end 47 tooutput the image signal thus obtained to a white balance correctionprocessing unit 92. The white balance correction processing unit 92serves to correct balance of respective colors on the basis of colortemperatuere of image signal delivered from the offset correctionprocessing unit 91 and differences between sensitivities of respectivefilters of the color filter 61 for four colors. A color signal acquiredafter undergone correction by the white balance correction processingunit 92 is outputted to a vertical direction simultaneousizing(synchronizing) processing unit 93. At the vertical directionsimultaneous-izing processing unit 93, delay elements are provided.Thus, shifts of time in vertical direction of signals outputted from thewhite balance correction processing unit 92 (hereinafter referred to asRG1G2B signals) are simulataneous-ized or synchronized (corrected).

A signal generation processing unit 94 performs interpolation processingfor interpolating color signal of 2×2 pixels of the minimum unit ofRG1G2B signals delivered from the vertical direction simulataneous-izingprocessing unit 93 into phase of the same space, noise removalprocessing for removing noise component of signal, filtering processingfor limiting signal band, and high frequency band correction processingfor correcting high frequency band component of signal band, etc. tooutput the RG1G2B signals thus obtained to a linear matrix processingunit 95.

The linear matrix processing unit 95 performs operation (computation) ofthe RG1G2B signals by the formula (1) on the basis of predeterminedlinear matrix coefficients (matrix of 3×4) to generate RGB signals ofthree colors.

$\begin{matrix}{\begin{bmatrix}R \\G \\B\end{bmatrix} = {\begin{bmatrix}a & b & c & d \\e & f & g & h \\i & j & k & l\end{bmatrix} \cdot \begin{bmatrix}R \\{G\; 1} \\{G\; 2} \\B\end{bmatrix}}} & (1)\end{matrix}$

An R signal generated by the linear matrix processing unit 95 isoutputted to a gamma-correction processing unit 96-1, a G signal isoutputted to a gamma correction processing unit 96-2, and a B signal isoutputted to a gamma-correction processing unit 96-3.

The gamma-correction processing units 96-1 to 96-3 perform gammacorrection with respect to respective signals of RGB signals which havebeen outputted from the linear matrix processing unit 95 to output RGBsignals thus acquired to a luminance (Y) signal generation processingunit 97 and a color difference (C) signal generation processing unit 98.

The luminance signal generation processing unit 97 synthesizes, at apredetermined synthesis ratio, RGB signals delivered from the gammacorrection processing units 96-1 to 96-3 in accordance with, e.g., theformula (2) to generate a luminance signal (Y).Y=0.2126R+0.7152G+0.0722B  (2)

A color difference signal generation processing unit 98 similarlysynthesizes, at a predetermined synthesis ratio, RGB signals deliveredfrom the gamma correction processing units 96-1 to 96-3 to generate acolor difference signal (C) to output the color difference signal (C)thus generated to a band limit thinning processing unit 99. The bandlimit thinning processing unit 99 generates color difference signals(Cb, Cr) on the basis of the color difference signal (C). It is to benoted that in a signal obtained by single chip 2×2 color coding, band ofcolor information generally does not exist to such a degree that suchband of color information exists in luminance signal. Accordingly, theband limit thinning processing unit 99 performs band limit processingand thinning processing with respect to color difference signal (C)delivered from the color signal generation processing unit 98 to therebyreduce color information data and to generate color difference signals(Cb, Cr).

A luminance signal (Y) generated by the luminance signal generationprocessing unit 97, and a color difference signal (C) generated by thecolor difference signal generation processing unit 98, or colordifference signals (Cb, Cr) generated by the band limit thinningprocessing unit 99 are outputted to the image monitor 50 through, e.g.,monitor interface 77 shown in FIG. 9.

In the image pick-up apparatus having the configuration as describedabove, in the case where photographing of image is instructed, themicrocomputer 41 controls TG46 to take image thereinto by image sensor45. Namely, rays of light of four colors are transmitted by color filter61 for four colors formed at the front face of image pick-up devicessuch as CCD, etc. (hereinafter referred to as CCD image pick-up device)constituting the image sensor 45. The rays of light thus transmitted aretaken in by the CCD image pick-up devices. The rays of light which havebeen taken in by the CCD image pick-up device are converted into colorsignals of four colors. The color signals thus obtained are outputted tothe front end 47.

The front end 47 implements correlating double sampling processing forremoving noise component, gain control processing, and digitalconvertion processing, etc. with respect to the color signal deliveredfrom the image sensor 45 to output image data thus obtained to thecamera system LSI 48.

At the signal processing unit 71 of the camera system LSI 48, offsetcomponent of color signal is removed by the offset correction processingunit 91, and balance of respective colors is corrected on the basis ofcolor temperature of image signal and differences between sensitivitiesof respective filters of the color filter 61 for four colors by thewhite balance correction processing unit 92.

Moreover, shifts of time in vertical direction of signals corrected bythe white balance correction processing unit 92 are simultaneous-ized orsynchronized (corrected) by the vertical direction simulataneous-izingprocessing unit 93, and interpolation processing for interpolating colorsignal of 2×2 pixels of the minimum unit of RG1G2B signals deliveredfrom the vertical direction simultaneousizing processing unit 93 intothe phase of the same space, noise removal processing for removing noisecomponent of signal, filtering processing for limiting signal band, andhigh frequency band correction processing for correcting high frequencycomponent of signal band, etc. are performed by the signal generationprocessing unit 94.

Further, at the linear matrix processing unit 95, signals generated bythe signal generation processing unit 94 (RG1G2B signals) aretransformed on the basis of predetermined linear matrix coefficients M(matrix of 3×4). Thus, RGB signals of three colors are generated. An Rsignal generated by the linear matrix processing unit 95 is outputted togamma-correction processing unit 96-1, a G signal is outputted to thegamma-correction processing unit 96-2, and a B signal is outputted tothe gamma-correction processing unit 96-3.

Gamma-correction is performed, by the gamma correction processing units96-1 to 96-3, with respect to respective signals of RGB signals obtainedby processing of the linear matrix processing unit 95. The RGB signalsthus acquired are outputted to the luminance signal generationprocessing unit 97 and the color difference signal generation unit 98.At the luminance signal generation processing unit 97 and the colordifference signal generation processing unit 98, respective signals of Rsignal, G signal and B signal which are delivered from thegamma-correction processing units 96-1 to 96-3 are synthesized at apredetermined synthesis ratio. Thus, luminance signal (Y) and colordifference signal (C) are generated. The luminance signal (Y) generatedby the luminance signal generation processing unit 97, and the colordifference signal (C) generated by the color difference signalgeneration processing 98 are outputted to the imagecompression/extraction unit 76 of FIG. 9, and are compressed by, e.g.,the JPEG format. The image data thus obtained after undergonecompression are outputted to the external memory medium 51 through themicrocomputer interface 73, and are stored therein.

Since one image data is formed on the basis of four kinds of colorsignals in a manner as stated above, the reproduction characteristicthereof results in that closer to seeing of the eye of the human being.

On the other hand, when reproduction (display) of image data stored inthe external memory medium 51 is instructed, image data stored in theexternal memory medium 51 is read out by the microcomputer 41. The imagedata which has been thus read out is outputted to the imagecompression/extraction unit 76 of the camera system LSI 48. At the imagecompression/extraction unit 76, compressed image data is expanded. Thus,image corresponding to obtained data is displayed on the image monitor50 through monitor interface 77.

Then, processing (procedure) for preparing the image pick-up apparatushaving the configuration as described above will be explained withreference to the flowchart shown in FIG. 11.

At step S1, processing for determining color filter for four colors,which determines spectral sensitivity characteristics of the colorfilter 61 for four colors provided at the image sensor 45 shown in FIG.5, is performed. At step S2, linear matrix coefficient M determinationprocessing for determining matrix coefficients M, which are set at thelinear matrix processing unit 95 shown in FIG. 10, is performed. Thedetail of the processing for determining color filter for four colors,which is executed at the step S1, will be described later with referenceto the flowchart shown in FIG. 12, and the detail of the processing fordetermining linear matrix coefficients M, which is executed at the stepS2, will be described later with reference to the flowchart shown inFIG. 16.

After color filter 61 for four colors is determined so that matrixcoefficients are determined, signal processing unit 71 shown in FIG. 10is prepared at step S3. Thus, processing proceeds to step S4, at whichcamera system LSI48 shown in FIG. 9 is prepared. Moreover, at step S5,the entirety of the image pick-up apparatus (e.g., digital camera) asshown in FIG. 5 is prepared. At step S6, evaluation of picture quality(“color reproduction characteristic”, “color discriminatingcharacteristic”) of the image pick-up apparatus prepared at the step S5is performed. Thus, processing is completed.

Here, object color referred in evaluating “color reproductioncharacteristic” and “color discriminating characteristic”, etc. will beexplained. The object color is calculated by value obtained byintegrating, within the range of visible light range (e.g., 400 to 700nm), product of “spectral reflection factor of object”, “spectral energydistribution of standard illumination”, and “spectral sensitivitydistribution (characteristic) of sensor for sensing object (colorfilter)”. Namely, the object color is calculated by the formula (3).Object color=k∫ _(vis)(spectral reflection factor of object)·(spectralenergy distribution of illumination)·(spectral sensitivity distributionof sensor for sensing object)dλ  (3)λ: wavelengthvis: visible light region (ordinarily 400 nm to 700 nm)

For example, in the case of observing a predetermined object by the eye,“spectral sensitivity characteristic of the sensor” of the formula (3)is represented by color matching function, and object colors of thatobject are represented by three stimulation values of X, Y, Z. Inconcrete terms, value of X is calculated by the formula (4-1), value ofY is calculated by the formula (4-2), and value of Z is calculated bythe formula (4-3). It is to be noted that values of constant k in theformulas (4-1) to (4-3) are calculated by the formula (4-4).X=k∫ _(vis) R(λ)·P(λ)· x (λ)dλ  (4-1)Y=k∫ _(vis) R(λ)·P(λ)· y (λ)dλ  (4-2)Z=k∫ _(vis) R(λ)·P(λ)· z (λ)dλ  (4-3)R(λ): spectral reflection factor of objectx(λ), y(λ), z(λ): color matching functionk=1/∫P(λ)· y (λ)dλ  (4-4)

Moreover, in the case where image of a predetermined object is takenthereinto by the image pick-up apparatus such as digital camera, etc.,“spectral sensitivity characteristic of sensor” of the formula (3) isrepresented by the spectral sensitivity characteristic of the colorfilter, and object color of that object of color value (RGB value(ternary), e.g., in the case of RGB filters. (three kinds)) iscalculated. In the case where RGB filter for detecting three kinds ofcolors is provided at the image pick-up apparatus, value of R iscalculated by the formula (5-1), value of G is calculated by the formula(5-2), and value of B is calculated by the formula (5-3) in practicalsense. Moreover, value of constant k_(r) in the formula (5-1) iscalculated by the formula (5-4), value of constant k_(g) in the formula(5-2) is calculated by the formula (5-5), and value of constant k_(b) inthe formula (5-3) is calculated by the formula (5-6).R=k _(r)∫_(vis) R(λ)·P(λ)· r (λ)dλ  (5-1)G=k _(g)∫_(vis) R(λ)·P(λ)· g (λ)dλ  (5-2)B=k _(b)∫_(vis) R(λ)·P(λ)· b (λ)dλ  (5-3)r(λ), g(λ), b(λ): spectral sensitivity distribution of color filterk _(r)=1/∫_(vis) P(λ)· r (λ)dλ  (5-4)k _(g)=1/∫_(vis) P(λ)· g (λ)dλ  (5-5)k _(b)=1/∫_(vis) P(λ)· b (λ)dλ  (5-6)

Then, the processing for determining color filter for four colors, whichis performed at the step S1 shown in FIG. 11, will be explained withreference to the flowchart shown in FIG. 12.

It is to be noted that while various methods exist as the method fordetermining color filter for four colors, explanation will be given inconnection with the processing in which, e.g., RGB filter is caused tobe base (one of existing (FIG. 1) G filters is caused to be G1 filter),and G2 filter for allowing color having high correlation with respect tocolor transmitted through the G1 filter to be transmitted therethroughis selected to supplement the G2 filter thus to determine color filterfor four colors.

At step S21, color target used for calculating UMG value is selected.For example, at step S21, there is selected color target in which manycolor patches representing existing colors are included and many colorpatches where importance is attached to memory color of the human being(skin color, green of plant, blue of sky, etc.) are included. As colortarget, there is, e.g., IT8.7, Macbeth Color Checker, GretargMacbethDigitalCamera Color Checker, CIE, Color Bar, etc.

Moreover, according to the object, color patch which can serve asstandard may be prepared from data such as SOCS (Standard Object ColorSpectra Database), etc. to use the color patch thus prepared. It is tobe noted that the detail of the SOCS is disclosed in ‘Johji Tajima,“Statistical Color Reproduction Evaluation by Standard Object ColorSpectra Database (SOCS)”, Color forum JAPAN 99’. Explanation will begiven below in connection with the case where Machbeth Color Checker isselected as color target.

At step S22, spectral sensitivity characteristic of G2 filter isdetermined. As the spectral sensitivity characteristic, there may beused spectral sensitivity characteristic which can be prepared fromexisting material, or there may be used spectral sensitivitycharacteristic obtained by assuming virtual curve C(λ) by cubic splinecurve (cubic spline function) as shown in FIG. 13 to change peak valueλ₀ of virtual curve C(λ), value w (value obtained by dividing sum of w₁and w₂ by 2) and value Δw (value obtained by dividing value obtained bysubtracting w₂ from w₁ by 2) within the range shown. It is to be notedthat values of w and Δw are set to values based on value of half-valuewidth. As a method of changing λ₀, w, Δw, those values are assumed to bechanged at pitch of e.g., 5 nm. The virtual curves C(λ) are representedby the following formulas (6-1) to (6-5) within respective ranges.

$\begin{matrix}{{C(\lambda)} = {{\frac{w_{2}^{3} + {3{w_{2}^{2}( {w_{2} - {{\lambda - \lambda_{0}}}} )}} + {3{w_{2}( {w_{2} - {{\lambda - \lambda_{0}}}} )}^{2}} - {3( {w_{2} - {{\lambda - \lambda_{0}}}} )^{3}}}{6w_{2}^{3}}\bigwedge 0} \leqq {\lambda - \lambda_{0}} \leqq w_{2}}} & ( {6\text{-}1} ) \\{{C(\lambda)} = {{\frac{w_{1}^{3} + {3{w_{1}^{2}( {w_{1} - {{\lambda - \lambda_{0}}}} )}} + {3{w_{1}( {w_{1} - {{\lambda - \lambda_{0}}}} )}^{2}} - {3( {w_{1} - {{\lambda - \lambda_{0}}}} )^{3}}}{6w_{2}^{3}}\bigwedge{- w_{1}}} \leqq {\lambda - \lambda_{0}} \leqq 0}} & ( {6\text{-}2} ) \\{{C(\lambda)} = {{\frac{( {{2w_{2}} - {{\lambda - \lambda_{0}}}} )^{3}}{6w_{2}^{3}}\bigwedge w_{2}} \leqq {\lambda - \lambda_{0}} \leqq {2w_{2}}}} & ( {6\text{-}3} ) \\{{C(\lambda)} = {{{\frac{( {{2w_{1}} - {{\lambda - \lambda_{0}}}} )^{3}}{6w_{1}^{3}}\bigwedge{- 2}}w_{1}} \leqq {\lambda - \lambda_{0}} \leqq {- w_{1}}}} & ( {6\text{-}4} ) \\{{C(\lambda)} = {{0\bigwedge{except}}\mspace{14mu}{for}\mspace{14mu}{the}\mspace{14mu}{above}\mspace{14mu}{range}}} & ( {6\text{-}5} )\end{matrix}$

It should be noted that although only filter G2 is supplemented in thisexample, only R filter and B filter of filter (R, G, G, B) shown in FIG.1 may be used to define the remaining two G1, G2 filters as virtualcurve of the above-mentioned formulas (6-1) to (6-5) in the vicinity ofgreen color. Moreover, similarly, only R and G, and only G and B may beused from the filter shown in FIG. 1. Further, among filters for fourcolors, filters for three colors may be defined as virtual curve, andfilters for four colors may be defined all as virtual curve.

At step S23, filter to be supplemented (G2 filter) and existing filters(R filter, G1 filter, B filter) are combined so that the minimum unit(set) of color filters for four colors is prepared. Moreover, at stepS24, UMG is used as filter evaluation coefficients with respect to colorfilter for four colors prepared at the step S23. Thus, UMG value iscalculated.

As explained with reference to FIG. 8, in the case where UMG is used,evaluation can be performed at a time with respect to respective colorfilters for four colors. Moreover, not only evaluation is performed bytaking into consideration spectral reflection factor of object, but alsoevaluation is made by taking into consideration reduction characteristicof noise. Since high evaluation is indicated with respect to filtershaving suitable overlap in spectral sensitivity characteristics ofrespective filters in the evaluation using UMG, it is possible tosuppress that high evaluation is indicated with respect to, e.g., filterhaving the characteristic in which the characteristic of R and thecharacteristic of G overlap with each other over a broad wavelength band(filter in which noise is amplified when respective color signals areseparated).

FIG. 14 is a view showing example of UMG value calculated at the colorfilter for three colors. For example, in the filter having thecharacteristic as shown in FIG. 14(A) in which characteristics of RGB donot overlap with each other, UMG value of “0.7942” is calculated. In thefilter having the characteristic as shown in FIG. 14(B) in whichcharacteristic of R and characteristic of G overlap with each other overa broad wavelength band, UMG value of “0.8211” is calculated. Moreover,in the filter having the characteristic as shown in FIG. 14(C) in whichrespective characteristics of RGB suitably overlap with each other, UMGvalue of “0.8879” is calculated. Namely, the highest evaluation isindicated with respect to the filter having the characteristic as shownin FIG. 14(C) in which respective characteristics of RGB suitablyoverlap with each other. This similarly applies also to color filter forfour colors. In this example, curve L31 shown in FIG. 14(A), curve L41shown in FIG. 14(B) and curve L51 shown in FIG. 14(C) represent spectralsensitivity of R, curve L32 shown in FIG. 14(A), curve L42 shown in FIG.14(B) and curve L52 shown in FIG. 14(C) represent spectral sensitivityof G, and curve L33 shown in FIG. 14(A), curve L43 shown in FIG. 14(B)and curve L53 shown in FIG. 14(C) represents spectral sensitivity of B.

At step S25, whether or not UMG value calculated at step S24 is “0.95”which is predetermined threshold value or more is judged. In the casewhere it is judged that the UMG value is less than “0.95”, processingproceeds to step S26 so that the prepared color filter for four colorsis rejected (is not used). At step S26, in the case where color filterfor four colors is rejected, the processing is completed thereafter(processing of the step S2 shown in FIG. 11 and those subsequent theretoare not executed).

On the other hand, at the step S25, in the case where it is judged thatUMG value calculated at the step S24 is “0.95” or more, that colorfilter for four colors is caused to be candidate filter used in thedigital camera at step S27.

Whether or not the color filter for four colors caused to be candidatefilter at the step S27 can be realized by existing material/dye isjudged at step S28. In the case where it is difficult to acquirematerial/dye, etc., it is judged that such color filter cannot berealized. Processing proceeds to step S26. As a result, the color filterfor four colors is rejected.

On the other hand, in the case where it is judged that material/dye,etc. can be acquired so that such color filter can be realized,processing proceeds to step S29. Thus, the color filter for four colorsthus prepared is determined as filter used in the digital camera.Thereafter, processing of step S2 shown in FIG. 11 and those subsequentthereto are executed.

FIG. 15 is a view showing an example of spectral sensitivitycharacteristic of color filter for four colors which has been determinedat step S29.

In FIG. 15, curve L61 represents spectral sensitivity of R, and curveL62 represents spectral sensitivity of G1. Moreover, curve L63represents spectral sensitivity of G2, and curve L64 represents spectralsensitivity of B. As shown in FIG. 15, the spectral sensitivity curve ofG2 (curve L63) has high correlation with respect to the spectralsensitivities curve of G1 (curve L62). Further, spectral sensitivity ofR, spectral sensitivity of G (G1, G2) and spectral sensitivity of Boverlap with each other within suitable range.

By utilizing the color filter for four colors determined in a manner asstated above, it becomes possible to improve particularly“discriminating characteristic of color” of “color reproductioncharacteristic”.

It is to be noted that it is preferable that filter having highcorrelation with respect to G filter of the existing RGB filter iscaused to be filter to be supplemented (G2 filter) in a manner as statedabove from viewpoint of utilization efficiency of light. In this case,it is desirable that peak value of the spectral sensitivity curve offilter to be supplemented experimentally exists within the range from495 to 535 nm (in the vicinity of peak value of the spectral sensitivitycurve of the existing G filter).

Moreover, in the case where filter having high correlation with respectto the existing G filter, since either one of two G filter constitutingthe minimum unit (R, G, G, B) shown in FIG. 1 is only caused to befilter of color to be supplemented to thereby have ability to preparecolor filter four colors, there is no necessity to greatly changeproduction process step.

In the case where color filter for four colors is prepared in a manneras stated above and the color filter thus prepared is provided at thedigital camera, since four kinds of color signals are delivered from thesignal generation processing unit 94 at the signal processing unit 71shown in FIG. 10, transform processing for generating signals of threecolors (R, G, B) from signals of four colors (R, G1, G2, B) is performedat the linear matrix processing unit 95. Since this transform processingis matrix processing with respect to luminous linear (luminance valuecan be represented by linear transform processing) input signal value,transform processing performed at the linear matrix processing unit 95will be called linear matrix processing hereinafter as occasion demands.

Then, the linear matrix coefficient M determination processing, which isexecuted at the step S2 shown in FIG. 11, will be explained withreference to the flowchart shown in FIG. 16. It is to be noted thatcolor target used in the linear matrix coefficient M determinationprocessing is caused to be Macbeth Color Checker, and the color filterfor four colors used is caused to have spectral sensitivitycharacteristic shown in FIG. 15.

At step S41, e.g., general day light D65 (illumination light L(λ)) usedas standard light source in CIE (Commision Internationaledel'Eclairange)is selected as illumination light. It is to be noted that theillumination light may be changed into illumination light, etc. of theenvironment where it is expected that the image processing apparatus isfrequently used. Further, in the case where plural illuminationenvironments assumed exist, it is conceivable to prepare plural linearmatrices. The case where the day light D65 is selected as illuminationlight will now be explained.

At step S42, reference values Xr, Yr, Zr are calculated. In concreteterms, the reference value Xr is calculated by the formula (7-1), thereference value Yr is calculated by the formula (7-2), and the referencevalue Zr is calculated by the formula (7-3).X _(r) =k∫ _(vis) R(λ)·L(λ)· x (λ)dλ  (7-1)Y _(r) =k∫ _(vis) R(λ)·L(λ)· y (λ)dλ  (7-2)Z _(r) =k∫ _(vis) R(λ)·L(λ)· z (λ)dλ  (7-3)R(λ): spectral reflection factor of objectx(λ), y(λ), z(λ): color matching function

Moreover, constant k is calculated by the formula (8).k=1/∫_(vis) L(λ)·y(λ)dλ  (8)

For example, in the case where color target is Macbeth Color Checker,reference values corresponding to 24 colors are calculated.

Then, at step S43, output values R_(f), G1 _(f), G2 _(f), B_(f) of colorfilters for four colors are calculated. In concrete terms, R_(f) iscalculated by the formula (9-1), G1 _(f) is calculated by the formula(9-2), G2 _(f) is calculated by the formula (9-3), and B_(f) iscalculated by the formula (9-4).R _(f) =k _(r)∫_(vis) R(λ)·L(λ)· r (λ)dλ  (9-1)G1_(f) =k _(g1)∫_(vis) R(λ)·L(λ)· g1(λ)dλ  (9-2)G2_(f) =k _(g2)∫_(vis) R(λ)·L(λ)· g2(λ)dλ  (9-3)B _(f) =k _(b)∫_(vis) R(λ)·L(λ)· b (λ)dλ  (9-4)r(λ), g1 (λ), g2 (λ), b(λ): spectral sensitivity distribution of colorfilter

Moreover, constant k_(r) is calculated by the formula (10-1), constantK_(g1) is calculated by the formula (10-2), constant K_(g2) iscalculated by the formula (10-3), and constant k_(b) is calculated bythe formula (10-4).k _(r)=1/∫_(vis) L(λ)· r (λ)dλ  (10-1)k _(g1)=1/∫_(vis) L(λ)· g1(λ)dλ  (10-2)k _(g2)=1/∫_(vis) L(λ)· g2(λ)dλ  (10-3)k _(b)=1/∫_(vis) L(λ)· b (λ)dλ  (10-4)

For example, in the case where color target is Macbeth Color Checker,output values R_(f), G1 _(f), G2 _(f), B_(f) corresponding to 24 colorsare calculated.

At step S44, matrix for performing transform processing to approximatefilter output value calculated at the step S43 into reference values(XYZ_(ref)) calculated at the step S42 is calculated by, e.g., errorleast square method in the XYZ color space.

For example, in the case where matrix of 3×4 to be calculated is assumedto be A represented by the formula (11), matrix transform (XYZ_(exp)) isrepresented by the following formula (12).

$\begin{matrix}{A = \begin{bmatrix}{a\; 0} & {a\; 2} & {a\; 3} & {a\; 4} \\{a\; 4} & {a\; 5} & {a\; 6} & {a\; 7} \\{a\; 8} & {a\; 9} & {a\; 10} & {a\; 11}\end{bmatrix}} & (11) \\{{{XYZ}\mspace{20mu}\exp} = {\begin{bmatrix}\hat{X} \\\hat{Y} \\\hat{Z}\end{bmatrix} = {\begin{bmatrix}{a\; 0} & {a\; 2} & {a\; 3} & {a\; 4} \\{a\; 4} & {a\; 5} & {a\; 6} & {a\; 7} \\{a\; 8} & {a\; 9} & {a\; 10} & {a\; 11}\end{bmatrix} \cdot \begin{bmatrix}R_{f} \\{G\; 1_{f}} \\{G\; 2_{f}} \\B_{f}\end{bmatrix}}}} & (12)\end{matrix}$

Moreover, square (E2) of error of matrix transform (formula (12)) withrespect to the reference value is represented by the following formula(13). On the basis of this value, matrix A which minimizes error ofmatrix transform with respect to reference value is calculated.E ² =|XYZref−XYZexp|²  (13)

Further, color space used in the error least square method may bechanged into color space except for XYZ color space. For example, theremay be performed transform processing into Lab, Luv, Lch color spaceseven with respect to sensory perception of the human being (perceptiveuniform color space) thereafter to perform similar operation, therebymaking it possible to calculate linear matrix which permits reproductionof color less in perceptive error. It is to be noted that since thesevalues of color space are calculated from XYZ values by non-lineartransform processing, non-linear calculation algorithm is used also inthe error least square method.

By operation as described above, matrix coefficients represented by theformula (14) are calculated as matrix coefficients with respect tofilter having spectral sensitivity characteristic shown in FIG. 15, forexample.

$\begin{matrix}{A = \begin{bmatrix}0.476 & 0.905 & 0.261 & {- 0.691} \\0.2 & 1.154 & {- 0.061} & {- 0.292} \\{- 0.004} & 0.148 & 0.148 & {- 0.481}\end{bmatrix}} & (14)\end{matrix}$

At step S45, linear matrix is determined. For example, in the case wherefinal RGB image data to be prepared is assumed to be represented by thefollowing formula (15), linear matrix (LinearM) is calculated in amanner as described below.RGBout=[R ₀ , G ₀ , B ₀]^(t)  (15)

Namely, in the case where illumination light is D65, transformexpression to transform sRGB color space into XYZ color space isrepresented by the formula (16) including ITU-R709.BT matrix, and theformula (17) is calculated by inverse matrix of the ITU-R709.BT matrix.

$\begin{matrix}{\begin{bmatrix}X \\Y \\Z\end{bmatrix} = {\begin{bmatrix}0.4124 & 0.3576 & 0.1805 \\0.2126 & 0.7152 & 0.0722 \\0.0193 & 0.1192 & 0.9505\end{bmatrix} \cdot \begin{bmatrix}R_{sRGB} \\G_{sRGB} \\B_{sRGB}\end{bmatrix}}} & (16) \\{\begin{bmatrix}R_{sRGB} \\G_{sRGB} \\B_{sRGB}\end{bmatrix} = {\begin{bmatrix}3.2406 & {- 1.5372} & {- 0.4986} \\{- 0.9689} & 1.8758 & 0.0415 \\0.0557 & {- 0.204} & 1.057\end{bmatrix} \cdot \begin{bmatrix}X \\Y \\Z\end{bmatrix}}} & (17)\end{matrix}$

By the matrix transform expression of the formula (12), and inversematrix of ITU-R709.BT matrix of the formulas (15) and (17), the formula(18) is calculated. In the right side of the formula (18), there areincluded inverse matrix of ITU-R709.BT matrix and linear matrix as valuemultiplied by the above-described matrix A.

$\begin{matrix}\begin{matrix}{\begin{bmatrix}R_{0} \\G_{0} \\B_{0}\end{bmatrix} = {\begin{bmatrix}3.2406 & {- 1.5372} & {- 0.4986} \\{- 0.9689} & 1.8758 & 0.0415 \\0.0557 & {- 0.204} & 1.057\end{bmatrix} \cdot}} \\{\begin{bmatrix}{a\; 0} & {a\; 2} & {a\; 3} & {a\; 4} \\{a\; 4} & {a\; 5} & {a\; 6} & {a\; 7} \\{a\; 8} & {a\; 9} & {a\; 10} & {a\; 11}\end{bmatrix} \cdot \;\begin{bmatrix}R_{f} \\{G\; 1_{f}} \\{G\; 2_{f}} \\B_{f}\end{bmatrix}}\end{matrix} & (18)\end{matrix}$

Namely, linear matrix (LinearM) of 3×4 is represented by the formula(19-1). Linear matrix with respect to color filter for four colorshaving spectral distribution characteristic shown in FIG. 15 in which,e.g., matrix coefficients of the formula (14) are used is represented bythe formula (19-2).

$\begin{matrix}{{LinearM} = \begin{bmatrix}{la} & {l\; 2} & {l\; 3} & {l\; 4} \\{l\; 4} & {l\; 5} & {l\; 6} & {l\; 7} \\{l\; 8} & {l\; 9} & {l\; 10} & {l\; 11}\end{bmatrix}} & ( {19\text{-}1} ) \\{\mspace{95mu}{= {\begin{bmatrix}3.2406 & {- 1.5372} & {- 0.4986} \\{- 0.9689} & 1.8758 & 0.0415 \\0.0557 & {- 0.204} & 1.057\end{bmatrix} \cdot}}} & \; \\{\mspace{124mu}\begin{bmatrix}{a\; 0} & {a\; 2} & {a\; 3} & {a\; 4} \\{a\; 4} & {a\; 5} & {a\; 6} & {a\; 7} \\{a\; 8} & {a\; 9} & {a\; 10} & {a\; 11}\end{bmatrix}} & \; \\{{LinearM} = \begin{bmatrix}1.238 & 1.084 & 0.228 & {- 1.55} \\{- 0.087} & 1.295 & {- 0.309} & 0.101 \\{- 0.018} & {- 0.029} & 1.535 & {- 0.485}\end{bmatrix}} & ( {19\text{-}2} )\end{matrix}$

The linear matrix calculated in a manner as described above is deliveredto linear matrix processing unit 95 shown in FIG. 10. Thus, since it ispossible to perform matrix processing with respect to signals (R, G1,G2, B) in which luminance can be represented by linear transform, it ispossible to reproduce color having higher fidelity from a viewpoint ofcolor engineering as compared to the case where matrix processing isperformed with respect to signal obtained after gamma processing isimplemented as in the case of processing at the signal processing unit11 shown in FIG. 2.

Then, evaluation performed at step S6 shown in FIG. 11 will beexplained.

In the case where comparison between color reproduction characteristicof the image pick-up apparatus provided with color filter for fourcolors having, e.g., spectral sensitivity characteristic shown in FIG.15 which have been prepared in a manner as stated above and colorreproduction characteristic of the image processing unit provided withcolor filter for three colors shown in FIG. 1 is performed, differenceas described below appears.

For example, color differences at Lab color space between output valueswhen image of Macbeth chart is picked up by two kinds of image inputapparatuses (image pick-up apparatus provided with color filter for fourcolors and image pick-up apparatus provided with color filter for threecolors) and reference values are respectively calculated by thefollowing formula (20).ΔE=√{square root over ((L ₁-L ₂)²+(a ₁-a ₂)²+(b ₁-b ₂)²)}{square rootover ((L ₁-L ₂)²+(a ₁-a ₂)²+(b ₁-b ₂)²)}{square root over ((L ₁-L ₂)²+(a₁-a ₂)²+(b ₁-b ₂)²)}  (20)

L₁-L₂ indicates lightness difference between two samples, and a₁-a₂,b₁-b₂ indicate component difference of hue/saturation of two samples.

FIG. 17 is a view showing calculation result by the formula (20). Asshown in FIG. 17, in the case of the image pick-up apparatus providedwith color filter for three colors, color difference is “3.32”, whereasin the case of the image pick-up apparatus provided with color filterfor four colors, color difference is “1.39”. “Way of seeing of color” atthe image pick-up apparatus provided with color filter for four colorsis more excellent (color difference is small).

In FIG. 18, R value of object R1 is set to “49.4”, G value thereof isset to “64.1” and B value thereof is set to “149.5”, and R value ofobject R2 is set to “66.0”, G value thereof is set to “63.7” and B valuethereof is set to “155.6”. Accordingly, in the color filter for fourcolors, RGB values of the object R1 and those of the object R2 becomevalues different from each other. Similarly to the case where object isseen by the eye, colors of respective objects are discriminated. Namely,filter capable of discriminating between four kinds of colors isprovided so that “discriminating characteristic of color” is improved.

While the color filter 61 for four colors is constituted in the aboveexample by arrangement such that B filters are provided at the left andright sides of G1 filter, and R filters are provided at left and rightsides of the G2 filter as shown in FIG. 6, such color filter may beconstituted by arrangement as shown in FIG. 19. In the color filter 61for four colors shown in FIG. 19, R filters are provided at left andright sides of the G1 filter, and B filters are provided at left andright sides of the G2 filter. Also by constituting the color filter 61for four colors in this way, it becomes possible to improve“discriminating characteristic of color”, “reproduction characteristicof color” and “reduction characteristic of noise” similarly to the colorfilter shown in FIG. 6.

Meanwhile, in the case where linear matrix coefficients M are determinedin such a manner to minimize color difference (ΔE value), if spectralsensitivities of color filters formed at the front stage portion of theimage sensor 45 overlap with each other as shown in FIG. 4, differencesbetween linear matrix coefficients M become large as indicated by theformula (21),

$\begin{matrix}{\begin{bmatrix}{r^{\prime}(\lambda)} \\{g^{\prime}(\lambda)} \\{b^{\prime}(\lambda)}\end{bmatrix} = {\begin{bmatrix}6.56 & {- 5.54} & 1.18 \\{- 2.01} & 3.12 & {- 0.16} \\0.12 & {- 0.28} & 1.07\end{bmatrix} \cdot \begin{bmatrix}{r(\lambda)} \\{g(\lambda)} \\{b(\lambda)}\end{bmatrix}}} & (21)\end{matrix}$

Since very small noise component is included in an output signal of theimage pick-up device, when such linear matrix coefficients M are used toperform color separation processing, very small noise component is alsoamplified. Accordingly, there takes place the necessity in which noisereduction characteristic is taken into consideration rather than colorreproduction characteristic so that differences between linear matrixcoefficients M do not large. However, there are instances where whenimage of object is actually picked up by such image pick-up apparatus,there is employed a method in which linear matrix coefficients M aredetermined under the condition where importance is attached to noisereduction characteristic rather than color reproduction characteristicby scene and/or environment to be imaged to adaptively perform linearmatrix processing so that improvement in picture quality can berealized. Conversely, there are instances where there is employed amethod in which importance is attached to color reproductioncharacteristic rather than noise reduction characteristic to determinelinear matrix coefficients M to adaptively perform linear matrixprocessing so that improvement in picture quality can be realized. Inaddition, since use purpose of the image pick-up apparatus is differentevery user, there are instances where user desires to arbitrarilyperform determination of linear matrix coefficients M.

In view of the above, in the image pick-up apparatus according to theinvention of this Application, in order to solve problems as describedabove, linear matrix coefficients M are determined in accordance withthe flowchart shown in FIG. 20.

First, determination of chart and illumination light which are used isperformed (step S50). Then, definition of color reproductioncharacteristic index ΔE(M) is performed (step S51). Then, definition ofnoise reduction index σN(M) is performed (step S52). Then, definition ofEEV(M) (Error Evaluation Value) is performed (step S53). Further, linearmatrix coefficients M are determined on the basis of the EEV (M) (stepS54). It is to be noted that, at the step S54, coefficients of EEV(M)are adaptively changed depending upon image pick-up condition, etc. todetermine corresponding linear matrix coefficients M. The detail ofrespective steps will be explained below.

Determination of chart and illumination light which are used (step S50)will be explained. In order to determine linear matrix coefficients M,it is necessary to determine color chart and light source forilluminating the color chart. As color chart, there are conceivablevarious reflection charts or transmission charts consisting of colorpatches having plural uniform color planes such as Macbeth ColorChecker, Digital Camera Color Checker, IT8.7, etc. As illuminationlight, there is conceivable illumination light (e.g., D55 light source,etc.) having spectral sensitivity close to light of the environmentwhere the image pick-up apparatus is frequently used. It is to be notedthat since it is conceivable that the mage pick-up apparatus is usedunder various light sources depending upon use purpose of user,illumination light is not limited to only light of the environment wherethe image pick-up apparatus is frequently used as illumination light.

Then, definition of color reproduction characteristic index ΔE(M) (stepS51) will be explained. The color reproduction characteristic is definedby difference between target color and color (hereinafter referred to asoutput color) that signal value in which linear matrix processing isperformed at linear matrix processing unit 95 of the image pick-upapparatus indicates. It is to be noted that while RGB values, YCbCrvalues or XYZ values, etc. are variously conceivable as value of color,a method of performing definition processing by using values(L*a*b*value, L*u*v*value, etc.) of color space where sensory perceptionis uniform with respect to seeing of the eye of the human being makes itpossible to more precisely indicate difference of color. For example,when target color of the k-th color patch in the color chart is assumedto be Lab_(ref) _(—) _(k)(L*_(ref) _(—) _(k), a*_(ref) _(—) _(k),b*_(ref) _(—) _(k)), and output color of the image pick-up apparatus isassumed to be L*a*b*_(shot) _(—) _(k) (L*_(shot) _(—) _(k), a*_(shot)_(—) _(k), b*_(shot) _(—) _(k)), color difference ΔE_(k) of this patchis represented by the formula (22).ΔE _(k)=√{square root over ((L* _(ref) _(—) _(k)-L* _(shot) _(—)_(k))²+(a* _(ref) _(—) _(k)-a* _(shot) _(—) _(k))²+(b* _(ref) _(—)_(k)-b* _(shot) _(—) _(k))²)}{square root over ((L* _(ref) _(—) _(k)-L*_(shot) _(—) _(k))²+(a* _(ref) _(—) _(k)-a* _(shot) _(—) _(k))²+(b*_(ref) _(—) _(k)-b* _(shot) _(—) _(k))²)}{square root over ((L* _(ref)_(—) _(k)-L* _(shot) _(—) _(k))²+(a* _(ref) _(—) _(k)-a* _(shot) _(—)_(k))²+(b* _(ref) _(—) _(k)-b* _(shot) _(—) _(k))²)}  (22)

Moreover, as color reproduction characteristic index ΔE(M), there areconceivable average ΔE value of respective patches of the color chart,and/or value in which weighting is performed with respect to respectivepatches so that importance is attached to color reproductioncharacteristic of a specific color, etc.

$\begin{matrix}{{\Delta\; E} = {\frac{1}{TotalPatchNum}\;{\int_{k = 1}^{TotalPatchNum}{{w_{k} \cdot \Delta}\; E_{k}}}}} & (23)\end{matrix}$

In the above formula, w_(k) indicates weighting coefficients withrespect to respective patches, and TotalPatchNum indicates the totalnumber of color patches.

Moreover, since linear matrix processing is implemented to output colorof the image pick-up apparatus in practice, L*a*b*_(shot) _(—) _(k)indicates function value of linear matrix coefficients M. Accordingly,ΔE_(k) and ΔE both result in function value of M.

$\begin{matrix}\begin{matrix}{{\Delta\;{E(M)}} = {\frac{1}{TotalPatchNum}\;{\int_{k = 1}^{TotalPatchNum}{{w_{k} \cdot \Delta}\;{E_{k}(M)}}}}} \\{= {\frac{1}{TotalPatchNum}\;{\int_{k = 1}^{TotalPatchNum}{w_{k} \cdot \sqrt{( {L_{ref\_ k}^{*} - {L_{shot\_ k}^{*}(M)}} )^{2} + ( {a_{ref\_ k}^{*} - {a_{shot\_ k}^{*}(M)}} )^{2} + ( {b_{ref\_ k}^{*} - {b_{shot\_ k}^{*}(M)}} )^{2}}}}}}\end{matrix} & (24)\end{matrix}$

Then, definition of noise reduction characteristic index σN(M) (stepS52) will be explained. The noise reduction characteristic index σN(M)is defined by standard deviation of signal values in which linear matrixprocessing is performed at linear matrix processing unit 95 of the imagepick-up apparatus. As signal value, there are conceivable RGB values,YCbCr values or XYZ values, etc. In this case, a method of definingsignal value by using values of color space where sensory perception isuniform with respect to seeing of the eye of the human being(L*a*b*value, L*u*v*value), etc. makes it possible to obtain noise valueσN_(k) having higher correlation with respect to noise feeling that thehuman being feels. At this time, noise value results in standarddeviation of respective components of color space in correspondence withcolor space of signal value. For example, in the case of RGB space,noise value results in σR, σG, σB. In the case of XYZ space, noise valueresults in σX, σY, σZ. In the definition of the noise reductioncharacteristic index σN(M), these noise values are used to determinesingle noise index. For example, in the case where image of a certaincolor patch is picked up under illumination light, noise value σN_(k) ofL*a*b* space is defined from values of lightness noise σL*_(k) and colornoises σa*_(k), σb*_(k) on the premise that lightness and color noiseare taken into consideration, as indicated by, e.g., the formula (25).

$\begin{matrix}{{\sigma\; N_{k}} = \sqrt{( \frac{\sigma\; L_{k}^{*}}{w\; L_{k}^{*}} )^{2} + ( \frac{\sigma\; a_{k}^{*}}{w\; a_{k}^{*}} )^{2} + ( \frac{\sigma\; b_{k}^{*}}{w\; b_{k}^{*}} )^{2}}} & (25)\end{matrix}$

In this case, wL*_(k), wa*_(k), wb*_(k) indicate weighting coefficientswith respect to respective standard deviation values, and are suitablyset by the correlation with respect to noise feeling that the eye of thehuman being feels. It is to be noted that noise value using variancevalue of other color space, etc. may be variously conceivable as noisevalue σN_(k). As noise reduction characteristic index σN(M), there areconceivable average σN value of respective patches of color chart,and/or value in which weighting is performed with respect to respectivepatches so that importance is attached to noise reduction characteristicof a specific color, etc.

$\begin{matrix}{{\sigma\; N} = {\frac{1}{TotalPatchNum}\;{\int_{k = 1}^{TotalPatchNum}{{w_{k} \cdot \sigma}\; N_{k}}}}} & (26)\end{matrix}$

In practice, since linear matrix processing is implemented to signalvalue of the image pick-up apparatus, σN_(k) and σN result in functionvalues of the linear matrix coefficients M.

$\begin{matrix}{{\sigma\;{N(M)}} = {\frac{1}{TotalPatchNum}\;{\int_{k = 1}^{TotalPatchNum}{{w_{k} \cdot \sigma}\;{N_{k}(M)}}}}} & (27)\end{matrix}$

Then, definition of EEV (M) (step S53) will be explained. By thedefinition of the above-described steps S51 and S52, EEV (ErrorEvaluation Value) (M) in which two values of color reproductioncharacteristic index ΔE(M) and noise reduction characteristic indexσN(M) which are function values of the linear matrix coefficients M aretaken into consideration is defined in a manner as indicated by theformula (28).EEV(M)=l[j{wc·h(ΔE(M))}+k{wn·i(σN(M))}]  (28)

In the above formula, h, I, j, k, l indicate function, wc indicatesweighting coefficient with respect to color difference, and wn indicatesweighting coefficient with respect to noise value. By changing wc and wnto determine linear matrix coefficients M so that EEV (M) becomesminimum, determination of linear matrix coefficients M in which bothcolor reproduction characteristic and noise reproduction characteristicare taken into consideration can be made. It is to be noted that it issufficient that in the case where importance is attached to colorreproduction characteristic, weighting is set so that wc>wn, and in thecase where importance is attached to noise reduction characteristic,weighting is set so that wc<wn.

Then, means for determining linear matrix coefficients M (step S54) willbe explained. The error least square method is applied to the EEV (M)defined by the step S53 to determine linear matrix coefficients M. Inthis case, at step S54, wc and wn are suitably determined to use, e.g.,Newton method, Steepest Descent method or Conjugate Gradient method,etc. as recursive algorithm and to apply error least square method, andthus to determine linear matrix coefficients M.

Moreover, at step S54, weighting coefficient wc with respect to colordifference and weighting coefficient wn with respect to noise value ofEEV (M) defined at the step S53 are adaptively changed by environmentand/or condition, etc. when image of object is picked up by the imagepick-up apparatus to determine linear matrix coefficients M by the errorleast square method. The state of change of noise value reductioncharacteristic index σN(M) when color reproduction characteristic indexΔE(M) is changed is shown in FIG. 21. As shown in FIG. 21, also in oneimage pick-up apparatus, trade-off of color reproduction characteristicindex ΔE(M) and noise reduction characteristic index σN(M) exists bylinear matrix coefficients M. By using this result, linear matrixcoefficients M are adaptively determined in accordance with variousimage pick-up environments and conditions, etc. Moreover, several setsof linear matrix coefficients M may be prepared in advance to allow userto select linear matrix coefficients M as occasion demands to adjustcolor reproduction characteristic index ΔE(M) and noise reductioncharacteristic index σN(M).

Here, a practical example in which in the case where the image pick-upapparatus includes CCD image pick-up device comprised of color filtersfor four colors having the characteristic as shown in FIG. 22, linearmatrix coefficients M are determined in accordance with theabove-described steps S50 to S54 will be described.

First, chart and illumination light which are used are determined (stepS50). As color chart, Macbeth Color Checker (including color patches of24 colors) is used. As illumination light, D55 light source (standardday light of 5500 k defined by CIE) is used. It is to be noted that itis assumed that respective spectral data are measured by using, e.g.,spectral radiation luminance meter.

Then, definition of color reproduction characteristic index ΔE(M) isperformed (step S51). Target color is caused to correspond to seeing ofthe eye of the human being, and color difference ΔE in Lab space iscaused to be index. In general, object color is defined, as indicated bythe formula (29), by value obtained by integrating product of “spectralreflection factor of object”, “spectral energy distribution ofillumination” and “spectral sensitivity distribution of sensor forsensing object” within the range of visible light region vis (ordinarily400 nm to 700 nm).Objectcolor=∫_(vis)(spectralreflectionfactorofobject)·(spectralluminanceof illumination)·(spectralsensitivity ofsensor for sensingobject)  (29)

Further, when color matching function representing spectral sensitivityof the eye of the human being is used, target colors XYZ_(ref) _(—)_(k)(X_(ref) _(—) _(k), Y_(ref) _(—) _(k), Z_(ref) _(—) _(k)) of thek-th patch of the color chart can be represented by the formula (30) byusing the formula (29).

$\begin{matrix} \begin{matrix}{X_{ref\_ k} = {m\;{\int_{vis}^{\;}{{{R_{k}(\lambda)} \cdot {L(\lambda)} \cdot {\overset{\_}{x}(\lambda)}}\;{\mathbb{d}\lambda}}}}} \\{Y_{ref\_ k} = {m\;{\int_{vis}^{\;}{{{R_{k}(\lambda)} \cdot {L(\lambda)} \cdot {\overset{\_}{y}(\lambda)}}\;{\mathbb{d}\lambda}}}}} \\{Z_{ref\_ k} = {m\;{\int_{vis}^{\;}{{{R_{k}(\lambda)} \cdot {L(\lambda)} \cdot {\overset{\_}{z}(\lambda)}}\;{\mathbb{d}\lambda}}}}}\end{matrix} \} & (30)\end{matrix}$In the above formula, R_(k)(λ): Spectralreflectionfactor of the k-thcolor patch within Macbethchart

-   -   L(λ): Illumination light D55 spectralradiationluminance    -   x, y, z: color matching function    -   m=1/∫_(vis)L(λ)· y(λ)dλ

Moreover, ordinarily, colors of the XYZ space are transformed intocolors of L*a*b* space by using the formula (31).

$\begin{matrix} \begin{matrix}{L^{*} = {{116 \cdot ( {Y/Y_{n}} )^{1/3}} - 16}} \\{a^{*} = {500 \cdot \{ {( {X/X_{n}} )^{1/3} - ( {Y/Y_{n}} )^{1/3}} \}}} \\{b^{*} = {200 \cdot \{ {( {Y/Y_{n}} )^{1/3} - ( {Z/Z_{n}} )^{1/3}} \}}}\end{matrix} \} & (31)\end{matrix}$

In the above formula, (X_(n), Y_(n), Z_(n)) indicates XYZ values ofcomplete diffused reflection surface (white point).

Further, target color XYZ_(ref) _(—) _(k) is transformed intoL*a*b*_(ref) _(—) _(k)(L*_(ref) _(—) _(k), a*_(ref) _(—) _(k), b*_(ref)_(—) _(k)) by using the formula (31).

In addition, raw data RGBX_(raw) _(—) _(k) (R_(raw) _(—) _(k), G_(raw)_(—) _(k), B_(raw) _(—) _(k), X_(raw) _(—) _(k)) which are signal valuesoutputted from the CCD image pick-up device are represented by theformula (32) by using the formula (29).

$\begin{matrix}\begin{matrix} \begin{matrix}{R_{raw\_ k} = {m\; r\;{\int_{vis}^{\;}{{{R_{k}(\lambda)} \cdot {L(\lambda)} \cdot {\overset{\_}{r}(\lambda)}}\;{\mathbb{d}\lambda}}}}} \\{G_{raw\_ k} = {m\; g\;{\int_{vis}^{\;}{{{R_{k}(\lambda)} \cdot {L(\lambda)} \cdot {\overset{\_}{g}(\lambda)}}\;{\mathbb{d}\lambda}}}}} \\{B_{raw\_ k} = {m\; b\;{\int_{vis}^{\;}{{{R_{k}(\lambda)} \cdot {L(\lambda)} \cdot {\overset{\_}{b}(\lambda)}}\;{\mathbb{d}\lambda}}}}}\end{matrix} \} \\{X_{raw\_ k} = {m\; x\;{\int_{vis}^{\;}{{{R_{k}(\lambda)} \cdot {L(\lambda)} \cdot {\overset{\_}{x}(\lambda)}}\;{\mathbb{d}\lambda}}}}}\end{matrix} & (32)\end{matrix}$In the above formula, r,g,b,x: spectralsensitivity distribution of CCDcolor filter of camera

-   -   mr=1/∫_(vis)L(λ)· r(λ)dλ    -   mg=1/∫_(vis)L(λ)· g(λ)dλ    -   mb=1/∫_(vis)L(λ)· b(λ)dλ    -   mx=1/∫_(vis)L(λ)· x(λ)dλ

Since the image pick-up apparatus performs linear matrix processing ofraw data RGBX_(raw) _(—) _(k)(R_(raw) _(—) _(k), G_(raw) _(—) _(k),B_(raw) _(—) _(k), X_(raw) _(—) _(k)) by using linear matrixcoefficients M (m0 to m11) at the linar matrix processing unit 95, imagepick-up data after undergone linear matrix processing is represented bythe formula (33).

$\begin{matrix}\begin{matrix}{\begin{bmatrix}R_{cam\_ k} \\G_{cam\_ k} \\B_{cam\_ k}\end{bmatrix} = {M \cdot \begin{bmatrix}R_{raw\_ k} \\G_{raw\_ k} \\B_{raw\_ k} \\X_{raw\_ k}\end{bmatrix}}} \\{= {\begin{bmatrix}{m\; 0} & {m\; 1} & {m\; 2} & {m\; 3} \\{m\; 4} & {m\; 5} & {m\; 6} & {m\; 7} \\{m\; 8} & {m\; 9} & {m\; 10} & {m\; 11}\end{bmatrix} \cdot \begin{bmatrix}R_{raw\_ k} \\G_{raw\_ k} \\B_{raw\_ k} \\X_{raw\_ k}\end{bmatrix}}}\end{matrix} & (33)\end{matrix}$

Further, the RGB_(cam) _(—) _(k) is transformed into XYZ value. It is tobe noted that 709 system matrix M₇₀₉ generally used is used as shown inthe formula (34) for the transform processing.

$\begin{matrix}\begin{matrix}{\begin{bmatrix}X \\Y \\Z\end{bmatrix} = {M_{709} \cdot \begin{bmatrix}R \\G \\B\end{bmatrix}}} \\{= {\begin{bmatrix}0.4124 & 0.3576 & 0.1805 \\{- 0.9689} & 1.8758 & 0.0415 \\0.0557 & {- 0.204} & 1.057\end{bmatrix} \cdot \begin{bmatrix}R \\G \\B\end{bmatrix}}}\end{matrix} & (34)\end{matrix}$

Then, X_(cam) _(—) _(k), Y_(cam) _(k) and Z_(cam) _(—) _(k) aredetermined by using the formula (34).

$\begin{matrix}{\begin{bmatrix}X_{cam\_ k} \\Y_{cam\_ k} \\Z_{cam\_ k}\end{bmatrix} = {\begin{bmatrix}0.4124 & 0.3576 & 0.1805 \\{- 0.9689} & 1.8758 & 0.0415 \\0.0557 & {- 0.204} & 1.057\end{bmatrix} \cdot \begin{bmatrix}R_{cam\_ k} \\G_{cam\_ k} \\B_{cam\_ k}\end{bmatrix}}} & (35)\end{matrix}$

Moreover, transform processing into L*a*b* value (L*a*b*_(cam) _(—)_(k)(L*_(cam) _(—) _(k), a*_(cam) _(—) _(k), b*_(cam) _(—) _(k)) isperformed by using the formula (31) to define color difference ΔE_(k) ofthe k-th patch of the color chart by the formula (36).ΔE _(k)=√{square root over ((L* _(ref) _(—) _(k)-L* _(cam) _(—)_(k))²+(a* _(ref) _(—) _(k)-a* _(cam) _(—) _(k))²+(b* _(ref) _(—)_(k)-b* _(cam) _(—) _(k))²)}{square root over ((L* _(ref) _(—) _(k)-L*_(cam) _(—) _(k))²+(a* _(ref) _(—) _(k)-a* _(cam) _(—) _(k))²+(b* _(ref)_(—) _(k)-b* _(cam) _(—) _(k))²)}{square root over ((L* _(ref) _(—)_(k)-L* _(cam) _(—) _(k))²+(a* _(ref) _(—) _(k)-a* _(cam) _(—)_(k))²+(b* _(ref) _(—) _(k)-b* _(cam) _(—) _(k))²)}  (36)

It is to be noted that since value of L*a*b*_(cam) _(—) _(k), is used,ΔEk can be represented as ΔE_(k)(M) because it is also function value ofthe linear matrix coefficients M. The color reproduction characteristicindex ΔE(M) is defined as average value of color differences ofrespective color patches as shown in the formula (37).

$\begin{matrix}{{\Delta\;{E(M)}} = {\frac{1}{TotalPatchNum}\;{\int_{k = 1}^{TotalPatchNum}{\Delta\;{E_{k}(M)}}}}} & (37)\end{matrix}$In the formula, TotalPatchNum=24: Total number of color patches

Then, definition of noise reduction characteristic index σN(M) isperformed (step S52). Here, noise reduction characteristic index σN(M)is defined on the basis of σL(M) component included in signal valueafter undergone linear matrix processing by the linear matrix processingunit 95 of the image pick-up apparatus.

In general, noise Noise_(—raw) included in signal CV_(—CCD) that the CCDimage pick-up device itself outputs is defined by the formula (38).

$\begin{matrix}\begin{matrix}{{Noise}_{\_ raw} = \sqrt{{{ShotNoiseCoef} \cdot {CV}_{\_ CCD}} + {DarkNoise}}} \\{= \sqrt{{ShotNoise} + {DarkNoise}}}\end{matrix} & (38)\end{matrix}$

It is to be noted that ShotNoiseCoef and DarkNoise are values determinedby the device characteristic of the CCD image pick-up device. DarkNoiserepresents noise component which is not dependent upon signal value(Fixed Pattern Noise, etc.), and ShotNoise represents noise componentdependent upon signal value (Sensor Dark Noise, Photon Shot Noise,etc.).

When the formula (31) is used, noise components included in raw data ofthe k-th color patch of the image pick-up apparatus to be evaluated aredefined by the formula (39).

$\begin{matrix} \begin{matrix}{{Noise}_{Rraw\_ k} = \sqrt{{{ShotNoiseCoef} \cdot R_{raw\_ k}} + {DarkNoise}}} \\{{Noise}_{Graw\_ k} = \sqrt{{{ShotNoiseCoef} \cdot G_{raw\_ k}} + {DarkNoise}}} \\{{Noise}_{Braw\_ k} = \sqrt{{{ShotNoiseCoef} \cdot B_{raw\_ k}} + {DarkNoise}}} \\{{Noise}_{Xraw\_ k} = \sqrt{{{ShotNoiseCoef} \cdot X_{raw\_ k}} + {DarkNoise}}}\end{matrix} \} & (39)\end{matrix}$

Moreover, in the literature (P. D. Burns and R. S. Berns, “ErrorPropagation Analysis in Color Measurament and Imaging”, Color Researchand Application, 1997), noise propagation theory as explained below isdescribed.

When a predetermined input signal X′=[x₁, x₂, . . . , x_(n)] is assumedto be linearly transformed into Y′=[y₁, y₂, . . . , y_(m)] by (m×n)matrix A, linearly transformed signal is represented by the formula(40).Y′=A·X′  (40)

When variance-covariance matrix Σ_(x) of input signal X′ is assumed tobe represented by the formula (41), diagonal component results in Noisevariance value of input signal.

$\begin{matrix}{\sum\limits_{x}\;{= \begin{bmatrix}\sigma_{11} & \sigma_{12} & \ldots & \sigma_{1n} \\\sigma_{21} & \sigma_{22} & \; & \; \\\vdots & \; & ⋰ & \; \\\sigma_{n1} & \; & \; & \sigma_{nn}\end{bmatrix}}} & (41)\end{matrix}$

If there is no correlation between input signal values mutually,covariance component (i.e., non-diagonal component) in the matrixcomponent becomes equal to zero. At this time, variance-covariancematrix Σ_(y) of output signal Y is defined by the formula (42).

$\begin{matrix}{\sum\limits_{y}\;{= {A \cdot {\sum\limits_{x}\; A^{\prime}}}}} & (42)\end{matrix}$

It is to be noted that the formula (42) results in propagation theoreticformula of Noise variance value between color spaces which can betransformed by linear transform.

Moreover, in order to transform signal RGB_(cam) _(—) _(k) afterundergone linear matrix processing by the linear matrix processing unit95 of the image pick-up apparatus into L*a*b*_(cam) _(—) _(k), it isnecessary to perform transform processing from RGB space to XYZ space(hereinafter referred to as RGB→XYZ transform processing) thereafter toperform transform processing from XYZ space to L*a*b*space (hereinafterreferred to as XYZ→L*a*b* transform processing). In the RGB→XYZtransform processing, it is possible to perform linear transform byusing 709 system M₇₀₉ shown in the formula (34), whereas in theXYZ→L*a*b* transform processing, it is necessary to perform non-lineartransform in a manner as shown in the formula (31). However, in theXYZ→L*a*b* transform processing, linear approximation can be made byusing Jacobian matrix J_(L*a*b*) _(—) _(k) because noise quantity isvery small. Thus, XYZ→L*a*b* transform processing can be performed bylinear transform in a manner similar to RGB→XYZ transform processing.

When value obtained by transforming signal value after linear matrixinto XYZ value is expressed as XYZ_(cam) _(—) _(k) (X_(cam) _(—) _(k),Y_(cam) _(—) _(k), Z_(cam) _(—) _(k)), this value can be represented bythe formula (43).J _(L*a*b*) _(—) _(k) =J ₀ D(XYZ _(cam) _(—) _(k))  (43)

In the above formula, J₀ is expressed as the formula (44).

$\begin{matrix}{{J_{0} = \begin{bmatrix}0 & 116 & 0 \\500 & {- 500} & 0 \\0 & 200 & {- 200}\end{bmatrix}}{\text{Moreover,~~when~~}{v( {a,b} )}\mspace{14mu}\text{is~~expressed~~as~~below,}}{{v( {a,b} )} = \{ {\begin{matrix}{\frac{1}{3} \cdot a^{- \frac{2}{3}} \cdot b^{- \frac{1}{3}}} & {\frac{a}{b} > 0.008856} \\{7.787 \cdot b^{- 1}} & {\frac{a}{b} \leq 0.008856}\end{matrix}\text{the~~formula~~(45)~~is~~provided.}} }} & (44) \\{{{D( {XYZ}_{cam\_ k} )} = \begin{bmatrix}{v( {X_{cam\_ k},X_{n}} )} & 0 & 0 \\0 & {v( {Y_{cam\_ k},Y_{n}} )} & 0 \\0 & 0 & {v( {Z_{cam\_ k},Z_{n}} )}\end{bmatrix}}{\text{In~~the~~above~~formula,~~}\text{XYZ}\text{~~value~~of~~complete~~diffused~~}}\text{reflection~~surface~~(white~~point)~~is~~set~~to}\mspace{40mu}{{XYZn}\mspace{14mu}{( {{Xn},{Yn},{Zn}} ).}}} & (45)\end{matrix}$

Accordingly, approximate matrix M_(total) _(—) _(k) for linearlytransforming raw data outputted from the CCD image pick-up device intoLab value is expressed as the formula (46).M _(total) _(—) _(k) =J _(L*a*b*) _(—) _(k) ·M ₇₀₉ ·M   (46)

When the matrix of the formula (46) and the formula of noise propagationtheory shown in formula (42) are applied, lightness noise σL_(k) at thek-th color patch can be calculated by the formula (47).

$\begin{matrix}{\sum\limits_{L^{*}a^{*}b^{*}{\_ k}}{= {{M_{total\_ k} \cdot {\sum\limits_{RGBraw\_ k}{\cdot {M_{total\_ k}^{\prime}\begin{bmatrix}{\sigma^{2}L_{k}^{*}} & {\sigma\; L^{*}a_{k}^{*}} & {\sigma\; L^{*}b_{k}^{*}} \\{\sigma\; a^{*}L_{k}^{*}} & {\sigma^{2}a_{k}^{*}} & {\sigma\; a^{*}b_{k}^{*}} \\{\sigma\; b^{*}L_{k}^{*}} & {\sigma\; b^{*}a_{k}^{*}} & {\sigma^{2}b_{k}^{*}}\end{bmatrix}}}}} = {M_{total\_ k} \cdot \lbrack { \quad\begin{matrix}{NoiseRawR\_ k} & 0 & 0 & 0 \\0 & {NoiseRawG\_ k} & 0 & 0 \\0 & 0 & {NoiseRawB\_ k} & 0 \\0 & 0 & 0 & {NoiseRawX\_ k}\end{matrix}\; \rbrack\mspace{11mu}{\quad{\quad{\cdot M_{total\_ k}^{\prime}}}}} }}}} & (47)\end{matrix}$

Accordingly, it is possible to derive the formula (48) from the formula(47).σL* _(k)=√{square root over (σ² L* _(k))}  (48)

It is to be noted that since the formula (48) is function of linearmatrix coefficients M, this formula can be represented by σL*_(k)(M).Since noise reduction characteristic index σN(M) is average value ofrespective lightness noises of color patch, σN(M) can be defined by theformula (49).

$\begin{matrix}{{{\sigma\;{N(M)}} = {( {1/24} ) \cdot {\int_{k = 1}^{24}{\sigma\;{L_{k}^{*}(M)}}}}}\ } & (49)\end{matrix}$

Then, EEV(M) in which color reproduction characteristic index ΔE(M) andnoise reduction characteristic index σN(M) which are defined asdescribed above are taken into consideration is defined by the formula(50) (step S53).EEV(M)=√{square root over ((wc·ΔE(M))²+(wn·σN(M))²)}{square root over((wc·ΔE(M))²+(wn·σN(M))²)}  (50)In the above formula,

-   wc: weighting coefficient with respect to color reproduction    characteristic-   wn: weighting coefficient with respect to noise reduction    characteristic.

Then, the formula (50) is solved by the error least square method todetermine linear matrix coefficients M. When, e.g., wc is set to 1, andwn is set to 2, EEV(M) is represented by the formula (51).EEV(M)=√{square root over (ΔE(M)²+(2·σN(M))²)}{square root over(ΔE(M)²+(2·σN(M))²)}  (51)

The formula (51) is solved by the error least square method to determinelinear matrix coefficients M by the formula (52) (step S54).

$\begin{matrix}{M = \begin{bmatrix}1.57 & {- 0.43} & {- 0.01} & {- 0.12} \\{- 0.14} & 1.25 & {- 0.37} & 0.26 \\{- 0.01} & {- 0.27} & 1.68 & {- 0.40}\end{bmatrix}} & (52)\end{matrix}$

On the other hand, color difference matrix in which wc is set to 1 andwn is set to 0, i.e., importance is attached (attention is paid) to onlycolor reproduction characteristic is represented by the formula (53)

$\begin{matrix}{M = \begin{bmatrix}1.48 & 0.56 & 0.35 & {- 1.39} \\{- 0.22} & 2.19 & {- 0.01} & {- 0.96} \\{- 0.06} & 0.27 & 1.93 & {- 1.14}\end{bmatrix}} & (53)\end{matrix}$

When comparison between the formula (52) and the formula (53) isperformed, it can be confirmed that difference between coefficients ofthe formula (53) is clearly larger, and the formula (53) is matrix toincrease noise.

Here, a practical example for adaptively determining linear matrixcoefficients M on the basis of setting of ISO sensitivity of the imagepick-up apparatus will be described. The image pick-up apparatus servesto amplify or attenuate signal inputted from the CCD image pick-updevice (hereinafter referred to as input signal) on the basis of settingof the ISO sensitivity. When setting of the ISO sensitivity of the imagepick-up apparatus is changed from ISO100 to ISO200, an input signal isamplified so that its value becomes equal to, e.g., value which is twicegreater than at the time of ISO100. The input signal thus amplified isinputted to the image pick-up apparatus. However, in the image pick-upapparatus, since linear matrix processing is performed by using the samelinear matrix coefficients M with respect to all input signals in spiteof setting state of the ISO sensitivity, in the case where setting ofthe ISO sensitivity is high, noise component included in an input signalis together amplified followed by amplification of the input signal.Accordingly, even when attempt is made to raise setting of the ISOsensitivity to obtain image of high resolution, an image in whichamplified noise component is included would be generated.

In view of the above, in the image pick-up apparatus according to thepresent invention of this Application, linear matrix coefficients M aredetermined by taking into consideration reduction characteristic ofnoise component included in an input signal amplified or attenuated onthe basis of setting of ISO sensitivity to perform linear matrixprocessing by using the linear matrix coefficients M. For example, asshown in the Table 1, weighting coefficient (wn) with respect to noisereduction characteristic is permitted to be changed in accordance withsetting of ISO sensitivity to substitute wc and wn into the formula (50)and to determine linear matrix coefficients M every setting of the ISOsensitivity. Accordingly, since the image pick-up apparatus can performlinear matrix processing by using linear matrix coefficients Mdetermined on the basis of setting state of the ISO sensitivity, even ifsetting of the ISO sensitivity is caused to be high, there is nopossibility that noise component is amplified followed by this, thus tohave ability to obtain image of high resolution.

TABLE 1 SETTING OF ISO SENSITIVITY wc wn 100 1 1 200 1 2 400 1 4 800 1 6

Here, an example for adaptively determining linear matrix coefficients Mon the basis of the environment where image of object is picked up bythe image pick-up apparatus will be described. For example, in the casewhere image of object, e.g., night view, etc. is picked by the imagepick-up apparatus, there are instances where the dark portion may occupythe large part of generated image. When noise takes place at that darkportion, such noise becomes very conspicuous. In such a case, it isbetter to take into consideration noise reduction characteristic ofnoise component rather than color reproduction characteristic.

In view of the above, in the image pick-up apparatus according to thepresent invention, linear matrix coefficients M are determined by takingnoise reduction characteristic and color reproduction characteristicinto consideration on the basis of scene in which image of object ispicked up to perform near matrix processing by using the linear matrixcoefficients M. For example, as shown in the histogram of FIG. 23, when70% of luminance component of generated image or more is included withinone half or less of the luminance dynamic range of the image pick-upapparatus, importance is attached to noise reduction characteristic todetermine linear matrix coefficients M. When except for the above, colorreproduction characteristic and noise reduction characteristic are takeninto consideration to determine linear matrix coefficients M. Inconcrete terms, as shown in the Table 2, weighting coefficient (wn) withrespect to noise reduction characteristic is permitted to be changed inaccordance with image pick-up scene to substitute wc and wn into theformula (50) to determine linear matrix coefficients M every imagepick-up scene. Accordingly, since the image pick-up apparatus canperform linear matrix processing by using linear matrix coefficients Mdetermined on the basis of image pick-up scene, even if dark portionoccupies the large part of generated image, it becomes possible toprevent noise component from becoming conspicuous.

TABLE 2 IMAGE PICK-UP SCENE wc wn NIGHT VIEW 1 8 OTHERS 1 2

Here, practical example for adaptively determining linear matrixcoefficients M on the basis of request of user who uses the imagepick-up apparatus will be described. There are many instances whereimage generated by picking up image of object by the image pick-upapparatus is required that noise is caused to be lesser as compared tocolor reproduction characteristic according to use purpose of user. Theuse purpose is the matter that manufacturing companies (makers) of theimage pick-up apparatus do not know, but is the fact that only usersknow.

In view of the above, the image pick-up apparatus according to thepresent invention of this Application serves to determine linear matrixcoefficients M on the basis of the condition that user intends, and toperform linear matrix processing by using the linear matrix coefficientsM. For example, as shown in the Table 3, weighting coefficient (wn) withrespect to noise reduction characteristic is permitted to be changed inaccordance with noise quantity adjustment variable to substitute wn andwc into the formula (50), to determine linear matrix coefficients Mevery noise quantity adjustment variable, and to store the linear matrixcoefficient M thus determined, whereby when user changes noise quantityadjustment variable through the interface of the image pick-upapparatus, predetermined linear matrix coefficients M are determined toperform linear matrix processing by using the linear matrix coefficientsM. Accordingly, since the image pick-up apparatus can perform linearmatrix processing by using the determined linear matrix coefficients Min accordance with request of user, it is possible to perform noisequantity adjustment corresponding to user situation of user

TABLE 3 NOISE QUANTITY ADJUSTMENT VARIABLE wc wn 0 1 1 1 1 2 2 1 3 3 1 44 1 5 5 1 6

It is to be noted that the Table 3 means that the larger becomes numericvalue of the noise quantity adjustment variable, the more noise quantityis reduced. Moreover, in place of performing setting of noise quantityadjustment variable before image pick-up operation, there may beemployed an approach in which when image pick-up start button of theimage pick-up apparatus is pushed down, image after undergone linearmatrix processing using several linear matrix coefficients M arepreserved (stored) into plural memories.

In the image pick-up apparatus constituted in this way, since colorfilter 61 for four colors formed at the front stage unit of the imagesensor 45 in accordance with the flowchart shown in FIG. 12, it ispossible to improve “discriminating characteristic of color” of “colorreproduction characteristic”. Moreover, since matrix processing isperformed with respect to signals (R, G1, G2, B) in which luminance canbe represented by linear transform at signal processing unit 71, colorhaving higher fidelity from a viewpoint of color engineering can bereproduced as compared to the case where matrix processing is performedwith respect to signal obtained after undergone gamma processing as inthe case of the processing at the signal processing unit 11 shown inFIG. 2. Further, since determination of linear matrix coefficients M isperformed in accordance with the image pick-up condition, etc. in thelinear matrix processing performed at the linear matrix processing unit95 of the signal processing unit 71, it is possible to improve colorreproduction characteristic and reduction characteristic of noise ofpicked up images.

It is to be noted that the present invention has been described inaccordance with certain preferred embodiments thereof illustrated in theaccompanying drawings and described in detail, it should be understoodby those ordinarily skilled in the art that the invention is not limitedto embodiments, but various modifications, alternative constructions orequivalents can be implemented without departing from the scope andspirit of the present invention as set forth by appended claims.

INDUSTRIAL APPLICABILITY

Since the image pick-up apparatus according to the present inventioncomprises adjustment unit for adjusting color reproduction valuerepresenting high fidelity reproduction of color with respect to way ofseeing of the eye of the human being and noise value representing noisefeeling that the human being feels, and matrix coefficient determinationunit for determining matrix coefficients on the basis of adjustment ofthe adjustment unit, and matrix transform processing unit for performingmatrix transform processing with respect to image which has been pickedup at image pick-up device unit provided at the image pick-up apparatuson the basis of the matrix coefficients, linear matrix coefficients Mare adaptively determined in accordance with image pick-up environmentand image pick-up condition, thus making it possible to perform linearmatrix processing by using the linear matrix coefficients M.

In addition, since the image pick-up method according to the presentinvention includes first step of adjusting color reproduction valuerepresenting high fidelity reproduction of color with respect to seeingof the eye of the human being and noise feeling that the human beingfeels, second step of determining matrix coefficients on the basis ofadjustment of the first step, and third step of performing matrixtransform processing with respect to image which has been picked up atimage pick-up device unit which serves to pick up image of object on thebasis of the matrix coefficients, linear matrix coefficients M areadaptively determined in accordance with image pick-up environment andimage pick-up condition, thus making it possible to perform linearmatrix processing by using the linear matrix coefficients M.

1. An image pick-up apparatus including an image pick-up device unitcomprised of color filters having different spectral characteristics,and serving to pick up an image of an object, the image pick-upapparatus comprising: adjustment means for adjusting color reproductionvalue and noise value representing noise; matrix coefficientdetermination means for determining matrix coefficients on the basis ofadjustment of the adjustment means; and matrix transform processingmeans for performing matrix transform processing with respect to animage which has been picked up at the image pick-up device unit on thebasis of the matrix coefficients, wherein the matrix coefficients aredetermined based on a color reproduction characteristic index and anoise reduction characteristic index.
 2. The image pick-up apparatus asset forth in claim 1, wherein the adjustment means serves to adaptivelyadjust the color reproduction value and the noise value in accordancewith image pick-up sensitivity of the image pick-up apparatus.
 3. Theimage pick-up apparatus as set forth in claim 1, wherein the adjustmentmeans serves to adaptively adjust the color reproduction value and thenoise value in accordance with the environment where image pick-upoperation is performed by the image pick-up apparatus.
 4. The imagepick-up apparatus as set forth in claim 1, wherein the adjustment meansserves to arbitrarily adjust the color reproduction value and the noisevalue.
 5. An image pick-up method of picking up an image of an object byan image pick-up apparatus including an image pick-up unit comprised ofcolor filters having different spectral characteristics, and serving topick up an image of an object, the image pick-up method including: afirst step of adjusting color reproduction value and noise valuerepresenting noise; a second step of determining matrix coefficients onthe basis of adjustment of the first step; and a third step ofperforming matrix transform processing with respect to an image whichhas been picked up at the image pick-up device unit on the basis of thematrix coefficients, wherein the matrix coefficients are determinedbased on a color reproduction characteristic index and a noise reductioncharacteristic index.